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(54) Motion image indexing system and method 

(57) A video source (101), sucfi as a VCR, plays 
back a nnotton image which Is input by an image input 
section (111). 

Partition of scenes and creation of images repre- 
senting scenes are performed by a catalog creation 
section (120). 

Additionally, at this time, an image representing the 
movement of an object appearing in the scenes is cre- 
ated. A representative image and a motion description 



image are initially stored in a catatog storage section 
(130) and then displayed in catalog form, by a display 
section (104) under control of a dnspiay control section 
(114). Since not only the representative image of 
scenes but also the movement of the object in those 
scenes are displayed, the contents of tfie scenes are 
easily grasped. 
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Description 

Background of the Invention 

Held of the Invention s 

The invention relates to a representative image dis- 
play ntethod and apparatus comprising of a series of 
multiple images into which a motion image is fonned, 
and which displays a representative image of each 10 
scene, and to motion image search apparatus which 
execute searches for a motion image using the repre- 
sentative image display apparatus. 

Description of Prior Art is 

Generally, a motion image contains multiple 
scenes. Each scene has independent meaning, and is 
defined by a group of consecutive images. Fig. 2 shows 
a situation where three scenes have been recorded on 20 
a video tape. Scene A is a view of a person, scene B is 
of a house, and scene C Is of an automobile. Each 
scene respectively comprises two (A1 and A2)» two (81 
and B2), and tiiree (C1-C3) images. A motion image 
search apparatus is used to quickly select a required 2s 
scene finom a motion image on recorded meda, such as 
vkleo tapes, when editing motion image inriages such as 
those required in a multimedia presentation creation or 
in broadcasting. Recently, apparatus have been devel- 
oped that display a representative image (hereinafter 30 
referred to as representative image) of each scene after 
a motion image has been partitioned into scene units, 
and that allow a desired scene to be selected using the 
representative image as an index. 

Fig. 3 is a block diagram from a catalog for "Media- 3S 
chef/CUT by Hitachi, Lid. showing a system using tiie 
product Details of its operation are given in "Media- 
chef/CUT fa Windows User Operation Manual" (Win- 
dows is a registered trademark). 

In the figure, the system comprises a PC 2 which 40 
performs general control, a video playt)ack apparatus 4 
which provides the motion image to the PC 2, and a 
color printer 6 which prints the edited images. The PC 2 
contains a video capture board 8 which captures and 
digitizes images provided from a video playback appa- 45 
ratus 4. This configuration performs, for example, the 
following steps. 

(1) First the motion image to be processed is 
loaded, (2) a transition point in the scene is automati- 
cally identified based on the degree of color change in so 
the images. (3) an image is decided to be a representa- 
tive Image that appears immediately after tiie transition 
point is observed, and (4) the representative image is 
stored or printed as an index for each scene. In the case 
of Fig. 2. a scene transition point is identified between ss 
A2 and B1. and between 82 and Cl. and tiie leading 
images of each scene, B1 and CI. become the repre- 
sentative images. 

This system helps the user in tape editing, for 



example, by printing out the representative images con- 
tained in one video tape cassette in a small sheet and 
affixing ttie sheet to the side of the video tape cassette. 
In addition, editing processes such as sorting, deletion, 
and trimming of ttie scenes are possible on ttie PC 2. 

In the system descrfl^ above, referendng the rep- 
resentative images alk)ws the user to look for desired 
scenes and replay ttiem. tiiat is. to search for a motion 
image. However, in representative images where a ship 
is afloat on the ocean, for example, the direction of the 
ship's movement can only be deternvned by reple^ing 
the motion image. If several representative images of 
this sort exist, for example, when searching for pictures 
where the ship moves to the right, tiie user must check 
each scene containing the representative image one by 
one. This manual search is not only laborious when 
there are a large number of similar representative 
images, but inconvenient when one tries to grasp the 
general movement of a motion image, and leaves much 
room for improvement for efficient editing. 

Summary of the Invention 

The inventbn takes tiiis problem into consideration 
and provides a representative image display method 
and apparatus, and motion image search apparatus, 
that enable the user to easily and intuitively grasp the 
contents of a motion image. 

The representative image display metiiod of the 
invention includes a scene partition process whk^ par- 
titions a motion image comprising a sequence of multi- 
ple images into scenes, a representative image 
selection process which selects a representative image 
with which to represent each partitioned scene, a transi- 
tion detection process which detects transitions occur- 
ring in the subject of each partitioned scene, a transition 
information hold process which associates and hokfe 
information oonceming detected transitions for every 
scene witfi the representative image of the corresporxi- 
Ing scene, and a display process which combines and 
displays tiie representative image of at least one scene 
witii transition information tiiat was associated witii tiie 
representative image. 

In this configuration, the motion image to be proc- 
essed is first input and ttien partitioned into scenes. 
Next, a representative image is selected to represent 
each partitioned scene. Furtiiermore. at tiiis time, 
changes occurring for an object within each scene are 
detected, and infbrrriation relating to the detected tran- 
sitions for each scene is held in association witii the 
scena Here "images'* refer to elements which comprise 
a motion image, for example frames or MPEG pictures. 
Hereinafter, they will also be referred to as frames. An 
"otDject** means anything v^ibie in the motion picture like 
a person. For example, in motion image, characters 
appearing on screen can be tiie objects. A "changes 
occuning for an objecT refers to, for example, a change 
due to movements of an object or a change in cdor of 
anot)ject. 
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Next the representative image of a scene and the 
transition Information associated with the representative 
image are combined and displayed. 

In accordance with the representative image dis- 
play method of the invention, the contents of a motion 
image can be easily and intuitiveiy grasped since infor- 
mation oonceming changes occurring in an object is 
displayed in addition to a representative image. 

On the other hand, the representative image dis- 
play apparatus of the invention includes an image input 
section which inputs and stores a motion image com- 
prising a sequence of muttipte images, a scene partition 
section which analyzes the input motion image, detects 
scene transition points, and partitions tiie motion image 
into scenes, a representative image selection section 
which selects a proper image as the representative 
image from among the images comprising each scene, 
a change information hold section which d^ects 
changes occurring in an object within a scene, and 
associates and holds tiie information concerning the 
change lor every scene with the representative image of 
the corresponding scene, and a display control section 
which combines and displays a representative image of 
at least one scene with the change information that was 
associated witii the representative image. 

In this configuration, the motion image which was 
input into tiie apparatus is first stored by the image input 
section. 

Next, tiie motion image is read, its scene transition 
points detected, and it is partitioned into scenes. Then, 
botii a representative image of each scene is aeated 
and the infamation on changes occunring in an object in 
every scene is associated arKl held with the representa- 
tive image of the corresponding scene. Rnaliy, a repre- 
sentative image of at least one scene and tiie change 
information that was associated with ttie representative 
image are combined and displayed. 

In accordance with the representative image dis- 
play apparatus of the invention, an effect similar to that 
of tiie above method is obtained. Furthemiore. tiie 
apparatus can be easily incorporated into the motion 
image search display apparatus. 

in one aspect of the representative image display 
apparatus of the invention, the changes occunring in the 
above object result from tiie movement of the object, 
and ttie above change detection section detects ttie 
movement of the object based on differences among 
multiple images. Naturally, the "movement of tiie objecT 
may occur as a result of movement at the camera. A 
"difference" between images is determined, for exam- 
ple, by obtaining ttie difference in pixel values at corre- 
sponding locations. On this assumption, the apparatus 
further includes a motion description image creation 
section which creates images visually showing move- 
ment of the object and the above display control section 
combines and displays ttie above representative image 
Witt) this motion description image. 

The apparatus enables ttie movement to be easily 
grasped in ttie ol^ect. 



In anottier aspect the invention, the above motion 
description image creation section fetches, at a preset 
time interval, an image stored in tiie image input section 
and aeates its outiine image. The outilne image is then 

5 superimposed onto a previously created motion 
description image so as to create a new motion descrip- 
tion Image, and the fetch, the creation of ttie outiine 
image and the superimposition processes are repeated 
to create new motion description images in succession. 

10 The "preset time interval" can be specified from an 
external source or can t>e the default value of ttie appa- 
ratus. In ttiis aspect the outline images at tiie various 
times are all shewn in one imaga Only the outline 
image that was created first may be used without 

IS change as a motion description image. 

According to this aspect of ttie invention, it is even 
easier to grasp the movement. 

In yet anottier aspect of ttie invention, when creat- 
ing outline images at preset time inten^te. ttie akx>ve 

20 motion description image creation section creates new 
motion description images wNIe changing the colors of 
tiie outlines t>ased on tiie elapsed time from a scene's 
start time. Here "color" includes black, white, and neu- 
tral tints (or achromatic shades). 

25 In this aspect, not only the movement but also the 
direction of movement can be easily grasped. 

In still another aspect of the invention, when an 
arbitrary time between the scene start time and end 
time is specified, the above display control section high- 

30 lights the location of the object at the specified time in 
the above motion description image. Since the "location 
of the subject" is indicated from the outiine image, one 
application of the aspect is to change the intensity of 
outiine image that corresponds to the specified time. 

35 According to this aspect, it t)ecomes easy to grasp 
tiie location of the object at a certain time, and ttits can 
facilitate ttie search for the search start position. 

The motion image search apparatus of the inven- 
tion performs searches for motion image and includes 

40 one of the ak)ove representative display apparatus, a 
video source which plays back and supplies a motion 
image to ttie atxive representative cfisplay apparatus, a 
video source control section which controls the play- 
back of tiie motion image by the video source, and a 

45 command input section which inputs commands for 
search processing. In ttiis configuration, when one 
scene and a playt)ack start time from the displayed 
scenes are specified by the command, the vkieo source 
control section fdkyws the command and performs play- 
so back of tiie motion image from tiie above playback start 
time of the specified scene. The playback enables the 
user to confirm the search result 

According to the motion Image search apparatus of 
the invention, all the advantages of the alxive represent- 

55 ative image display apparatus are obtainat>le and the 
motion image editing efficiency is greatty improved. 
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Brief Description of the Drawings 

Fig. 1 is an overall block diagram of the motion 
image search apparatus concerning embodiment 1. 

Rg. 2 exemplifies a state where three scenes have s 
been recorded on video tape. 

Rg. 3 is a block diagram, taken from a catalog for 
Medlachef/CUX of a system using "Mediachef/CLTT. 

Rg. 4 is a block diagram of motion description 
image aeation section 1 23 of embodiment 1 . to 

Fig. 5 is a flowchart showing the f k)w of processes 
executed by the motron description creatk)n section 123 
of embodiment 1. 

Fig. 6 shows a Ul which is displayed at display sec- 
tion 1 04 of emlxxJiment 1 . is 

Fig. 7 is a f bwchart showing the flow of processes 
which use the Ul of Rg. 6. 

Fig. 8 is an enlargement of the motion description 
image display area included in Fig. 6. 

Rg. 9 is a block diagram of a motion descrqptk)n 20 
image aeation section of embodiment 2. 

Rg. 10 is a flowchart showing the flow of processes 
executed by the motion description image creation sec- 
tion 123 of embodiment 2. 

Fig. 1 1 shows a result obtained when the motion 25 
description image of Fig. 8 is recreated by the appara- 
tus of embodiment 2. 

Rg. 12 is a block diagram of a catalog creation sec- 
tion 120 of embodiment 3. 

Fig. 1 3 is a block diagram of a catalog storage sec- 30 
tkm 130 of embocfiment 3. 

Fig. 14 shows the Ul which is displayed at display 
section 104 of entxxjiment 3. 

Description of Preferred Embodiments 3S 

Here, preferred enlbodiments of a rep-esentative 
image dsplay apparatus and a motion image search 
apparatus incorporating the representative image dis- 
play apparatus are described. 

This description will also clarify the representative 
image display method of the invention. In the motkm 
image search apparatus described in the emtxxfiments 
below, a playback apparatus which plays t>ack a video 
medium is included, and the playt>ack apparatus is elec- 45 
trically controlled directly from a main control section in 
the motion image search apparatus. Although this con- 
figuration is ideal, Ibr example, in broadcasting where 
relatively advanced functbns are required, its applica- 
tions are of course not limited thereto. so 

Embodiment 1 

In tiiis embodiment, an apparatus is descrit>ed 
which creates an Image showing the motion of an object ss 
in every scene and displays it together with a represent- 
ative image. 



Configuration 

Rg. 1 is an overall block diagram of the motion 
image search apparatus concerning this embodiment. 
Since a main control section 110 controls the overall 
apparatus in this figure, this and a command input sec- 
tion 103 mentioned later oonrespond to ttie representa- 
tive image display apparatus. A vkleo source 101 plays 
back recorded video media such as those for VCRs and 
provides motion image to the main control section 1 10 
and a motion image display section 102. The main con- 
trol section 1 10 creates representative images and the 
motion description images from the provided motion 
image, and motion image display section 102 simply 
displays the provided motion Inrage. The motion image 
display section 102, for example, a monitor which (fis- 
plays ordinary NTSC format signals, allows a user to 
directiy view the searched motion image. 

A video source control section 112 controls the 
playt)ack of motion image from a video source 101. It 
performs control functions required for the search proc- 
ess, such as "stop after 10 seconds from tiie start", in 
addition to the various control functions (tape rewind, 
fast fonvard, pause, and so fbrtii) available on the front 
panel of home VCRs. Control signals are connected to 
the video source 101 by a cable. A video source control 
section 1 12 controls the playback of the nrK>tion image 
that is necessary for the main control section 1 10 to cre- 
ate a representative . image, and necessary for the 
motion display section 102 to display. 

The played back motbn image is Input by an image 
input section 111. Image input section 111 includes an 
A/D converter and a frame memory (not shown). Image 
input section 1 1 1 digitizes ttie Input image, stores it into 
the frame memory and waits for further processing. The 
image input section 1 1 1 may also contain a disk storage 
apparatus which secondarily stores the contents of the 
frame n^mory. 

In this case, it becomes unnecessary to read the 
motion image required for processing every time from 
video source 101, and processing efficiency can be 
improved. 

A catalog creation section 120 is composed of a 
series of software modules which read Image data from 
tiie frame memory, and creates a catalog biased on this 
data, and which conprises a scene partition section 
121, a representative Image selection section 122. and 
a motion description image aeation section 123. A cat- 
alog is a list of informatbn concerning multiple scenes. 
On the other hand, based on ttie contents of the data 
output from the catalog creation section 120, a catalog 
storage section 1 30 sorts and stores the data, and com- 
prises a scene partition data storage section 1 31 . a rep- 
resentative image storage section 132, and a nxition 
description image storage section 133. 

The scene partition section 121 analyzes the digi- 
tized image using known metiiods and partitions the 
motion image into scenes based on ttie difference 
between two consecutive scenes. Based on ttie position 
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(for example a time code or counter value such as LTC 
or VITC recorded on a tape) on the medium at the par- 
tition point, the start point and end point of each scene 
are stored in a scene partition data storage section 1 31 . 

The representative image selection section 122 5 
selects a certain frame from the scene as a representa- 
tive image. The representative image may k>e a frame at 
the beginning, middle, or end of the scene, or a frame 
specified by the user. To identify the scene which the 
representative image represents, information such a 
scene number is stored into the representative image 
storage section 132. 

Motion description image creation section 123 per- 
forms a distinctive process in this embodiment. Fig. 4 is 
a block diagram of a motion description image creation 
section 123. 

In this figure, an outiine image creation section 201 
processes tiie input image and creates an outiine 
image. A composite intage creation section 202 per- 
forms composite processing on the created outiine 
image and an outiine image storage section 203 tempo- 
rarily stores tiie composed outiine image. In this embod- 
imerrt tiie outiine of an object included in a given image 
within a scene is sampled in each image, and these out- 
lines are superimposed to aeate one image. 

Command input section 103 permits the entry of 
cormiand inputs by the user and employs a device such 
as a keytx>ard. mouse, or various pointing devices. The 
input command is processed by an input management 
section 113. which is a command dispatcher, to initiate 
the required action wittiin tiie main control section 110. 

A display control section. 114 includes a system 
program for the display system which centrally controls 
the cfisplay of various images such as representative 
images and motion description images, and a display 
circuit such as a VGA controller and its driver. In tiie dis- 
play control section 114. the display state such as for 
the location of a catalog which is read from catalog stor- 
age section 130 is ultimately determined and displayed 
at a display section 104. The display section 104 may 
be an ordinary display apparatus on which represenfo- 
tive Images and nrxytion description images are dis- 
played. 

Process Operations 

The main conbroi section 110, which receives a 
scene partition start instruction based on command 
inputs, issues a play-back instruction to tiie video 
source 101 through the video source control section 
112. The motion image output from video source 101 is 
displayed by the motion image display section 102 and 
digitized in frame units tTy the image input section 111. 

Next each frame is sent to a catalog aeation sec- 
tion 120. At the scene partition section 121 , a histogram 
is taken for the input image and its preceding image. 
aruJ a scene's partition point is judged when the abso- 
lute value of its difference reaches or exceeds a ttiresh- 
old. This method itself is known. Instead of fliis metiicd. 



any characteristic physical parameter between consec- 
utive images may be compared to detect tiie partition 
point Once the partition point is ascertained, informa- 
tion such as a time code recorded on the video tape is 
acquired from tiie video source 101 and stored in the 
scene partition data storage section 1 31 . In parallel with 
tills, a representative image is created and stored as a 
still image file in the representative image storage sec- 
tion 132. Next the motion description image is created. 

Rg. 5 is a flowchart showing the flow of processes 
by motfon descr^'on image creation section 123. 
According to this figure, tiie sent image is passed 
through a Compass-gradient filter In the outiine Image 
aeation section 201 and an outiine image is created 

(5211) . In tiie outiine image, each pixel takes on one of 
two values 0 and 1 . where pixels on ttie outiine become 
1 and other pixels become 0. Outiine images may be 
obtained at a user specified frame interval, for example, 
every 10 frames, or at an interval preset as a default 
value in the apparatus. 

Next, the image which has been stored in the out- 
line Image storage section 203 is read and composed 

(5212) by ttie composite image creation section 202 
with the outiine image created in 8211. Composing of 
tiie Image is achieved by taking the logical OR of pixel 
values at the same coordinates for all pixels in the two 
images. The composite image tiius created is stored 

(5213) in tfie outline image storage section 203. These 
processes are repeated until a scene partition point is 
detected. When the scene partitfon point is detected, 
the image stored in the outiine image storage section 
203 is ti'ansferred to and stored in the motion descrip- 
tion image storage section 133. 

Here, the above processes are described together 
wrtii user input operations and actual image display 
examples. 

Rg. 6 shows a user interface (UQ displayed by the 
display section 104, and Fig. 7 is a flowchart showing 
ttie flow of processes using the Ul. In Rg. 6. a scene 
partition start button 152 starts scene partition. A play- 
back button 153 initiates playback of the motion image 
for the selected scene and a stop txitton 154 stops the 
motion image being played back. In Rg. 6. tiie entire 
screen above these buttons is a catalog display area 
151 which is organized into 3 scene display areas 155. 
each conresponding to one scene. Each scene cfisplay 
area 155 contains a representative image display area 
156 whfoh displays the representative Image of tiiat 
scene, a motfon descriptfon image display area 157 
which displays ttie motion description image, arid a 
scene partition data display area 158 which displays 
each scene*^ start point (start time) and end point (end 
time). The first two digits of the time display indicate the 
hours, followed by minutes, seconds, and frame 
number. The frame number for tiie NTSC format is from 
0 to 29 since the number of frames displayed is 30 per 
second. When playing back a motion image, tiie user is 
allowed to specify beforehand how.much to play l>ack 
(that is. for how much of the motion image scene parti- 
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tion is to be performed). This specified amount is con- 
verted to a time code on the rriedtum and held hi the 
apparatus as the specified end position. 

In the Ul, once the user presses the scene partition 
start button 152. the input management section 113 
interprets this command (SI 70) and a playback instruc- 
tion (SI 71) is sent to the video source 101 . The medium 
in the video source 101 is played back (SI 72) and the 
motion image is sent (SI 73) to the catalog creation sec- 
tion 120. At this time; it is judged whether the time code 
on the medium indicates the specified end position 
(SI 74) or the status indicates the end position (SI 75) of 
the medium, and if either is realized the process ends 
(SI 76). 

If the process does not end (No at SI 75), an outline 
image for that frame is created and a motion desaiption 
image is created (SI 77). In parallel with this, the input 
image Is judged (SI 78) as to whether It should be a rep- 
resentative Image, and a representative image is cre- 
ated (SI 79), for example, when a frame specified 
beforehand by tiie user Is to be processed. 

Next, it is judged (SI 80) whether the scene parti- 
tion point is reached, and if It is, tiie motion description 
image is transfened (SI 82) to tiie motion desaiption 
image storage section 1 32, and catalog display intorma- 
tfon is created (S183) to display (S184) scene partition 
data, tiie representative image, and the motion descrip- 
tion image. Then the process ends. If the scene parti- 
tion point Is not reached, the next frame is read and the 
same process Is repeated. 

From the steps above, a scene's representative 
image and motion description image are displayed on a 
display apparatus. An enlarged cGagram of the motion 
description Image display area included in Fig. 6 Is 
shown in Fig. 8, where tiie movement of the object 
extending over multiple images is expressed consecu- 
tively in one image. As a result, the user can understand 
the scene in general from the representative image, ard 
in detail from the motion description image. Based on 
these types of information, the user can actually view 
the scene by selecting it on the saeen, by using a 
mouse, for example. As shown in Rg. 6. the selected 
scene has its area frame highlighted. The above txjttons 
become effective tor the selected scene. TTiat is, the 
playback button 153 enables playt>ack of the scene and 
the stop button 154 enables playback to be stopped. In 
this manner, a highly eff ident motion image search is 
achieved. The above Is an overview of the embodiment 
Possible improvements and variations for this embodi- 
ment are discussed in the following. 

(1) Display the motion description infamation as 
characters, symbols, and so forth. 

Instead of using a motion description image, 
the direction of movement of the subject can be 
expressed using characters such as "right" and 
"up", or using symbols such as "^" and "1^. This 
can t>e achieved by judging the angle of tiie sub- 
ject's movement vector. When expressing move- 



ment using characters, symbols, and so fortii. the 
processing time and menwry capacity required for 
tiie image aeation can be reduced. 
(2) Display the transition information otiier tiian 

5 movement 

Whea for example, the ctoieO, is a neon sign, 
the object itself does not move but its color 
changes. In this case, examining such differences 
as color or brightness In the object before and after 

10 the cliange allow information such as ^becomes 
red", "becomes white", or "t)eoomes bright" to be 
displayed as text data beside the representative 
image. 

IS Embodiment 2 

Although the motion description image was created 
In emtxxliment 1 by superimposing successive outline 
images, the superimposition in embodiment 2 is per- 
20 formed while varying the color of tiie outline image. For 
this reason, if the outiine image concerning a subse- 
quentiy input frame is displayed using a darker cofor. for 
example. It becomes easier to grasp the movement of 
tiie display. 

25 Rg. 9 is a block diagram of tiie motion description 
image creation section 123 in tiiis emtxxiiment The dif- 
ference between tNs figure and Rg. 4 is that a hue set- 
ting section 301 is added. Hue setting section 301 
varies tiie hue of the outiine image with the lapse of 

30 time. 

Fig. 1 0 is a flowchart showing tiie f fow of processes 
executed by tiie motion description image creation sec- 
tion 1 23. The difference between this figure and Fig. 5 is 
that S312 is Inserted after S21 1 . That is. after an outiine 

35 Image Is created (S211). the outilne's hue is varied 
(S312) by tiie hue setting section 301 with the lapse of 
time from tiie start time of the scene. After this, tiie proc- 
ess is repeated as shown In Rg. 5 and the motion 
description image Is created. 

40 Gradation processing in which only the brightness 
is decreased can be employed in the processing of hue 
setting section 301 . In this case, tiie outiine pixel values 
for the outiine image obtained the first time are set to a 
maximum value of 255 on an 8-bit gray scale and 

45 decreased by 5 at a time, for exanple. in sut)sequently 
input images. Rg. 1 1 is an image obtained as a result of 
reK»-eating the motion description im^e of Rg. 8 using 
tiie apparatus of tiiis embodiment, and it becomes eas- 
ier to grasp tiie movement of the subject. 

so 

Embodiments 

In this embodiment the input of an art^rary time 
between the start time and end time of a scene high- 
55 lights the outiine image corresponding to the input time 
(hereinafter referred to as specified time) in the motion 
description image fbr tiie scene. Furthermore, the 
medium is actually forwarded or reversed until the posi- 
tion where the image corresponding to the specified 
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time is found. 

Rg. 12 is a block diagram of a catalog creation sec- 
tion 120 In this embodiment and Fig. 13 is a block dia- 
gram of a catalog storage section 130. A new block 
added in Fig. 12 is a time calculation section 401 which 
obtains the amount of change in time occun^ing when a 
time button described later is pressed after the specified 
time was input for a scene. 

On the other hand, a new block in Fig. 13 is a time 
storage section 41 1 which stores, for every scene, tiie 
time calculated by the time calculation section 401 . The 
time storage sectfon 41 1 Is a tnjffer which can store at 
least one time value for every scene, and is useful when 
a search temporarily moves to another scene and then 
returns back to the original scene, so that the user can 
determine wtnich image had been referenced prevfously. 

Fig. 14 shows the Ul displayed by the display sec- 
tion 104. 

The difference between this figure and Fig. 6 is that 
a display of "cun^ent time" is newly added. When a spec- 
ified time is input, the current time is initialized with tNs 
value. Subsequentiy, every time a time button is 
pressed, the current time increases or decreases. Time 
buttons comprise a previous time button 451 which 
specifies a previous time and a sut>sequent time button 
452 whicii specifies a 8ut>sequent time. 

Using tiie configuration above, tiie user first selects 
a scene. Here, the topmost scene is selected. Next, tiie 
specified time is entered, for example, through a key- 
board. 

In the figure, tiie allowable input is in tiie range of 
scene start time 00.-00:00:00 to end time OO.D2.'05:20. 
For example, when 00:01 :02:28 is specified, tiie outiine 
image with the dosest time is highlighted. Images are 
input in 1/30 second units, and when an outiine Image is 
created for every image, for example, tiie time corre- 
sponding to the outiine Image (called an outiine Image 
time) takes on a disaete value in 1/15 second units. For 
this reason, the specified time and the outiine image 
time usually do not match exactly, so the closest time is 
selected. The selected outline image time is stored in a 
time storage section 411. 

In this embodiment along with highlighting the out- 
line image, the playback position of the actual medium 
is fonvarded or rewound to the outiine image time. Thus, 
the object on the medium appears at the position of the 
highlighted outline image. As a result the user can see 
where the object is located at the specified time, and if 
playt>ack from that position is desired, tiiis can be 
achieved simpiy by issuing a playk>ack instruction. 

When the position at the specified time is different 
from where playback is to start, tiie above time buttons 
are pressed. The pressing of the button is judged by the 
input management section 113 and a new cun^ent time 
is calculated by the time calculation section 401. Since 
the outiine image time takes on a disaete value in 1/15 
second units, the current time increases or decreases 
by 1/15 second each time tiie t)utton is pressed, and the 
outiine image adjacent to the currentiy highlighted out- 



line image is highlighted. The piayt>ack position on tiie 
medium is also precisely adjusted. 

The contents of time storage section 411 are 
updated with the new cun^ent time. From the processes 

5 above, it becomes possible to set the start position of 
playl>ack with extremely high precision. If tiie sut)se- 
quent time button 452 is pressed at the final outiine 
image time of tiie scene, or conversely, if the previous 
time button 451 is pressed at the initial outiine image 

10 time, tiiese instructions will be ignored. 

In this embodiment, it is necessary to consider tiiat 
the motion description image Is only one Image. That is. 
once this image is created, it becomes necessary to 
add Information to the motion description Image regard- 

75 ing which outiine con^esponds to what time. For this rea- 
son, (1) motion description images having differing 
levels of brightness according to preset times are ae- 
ated using tiie method of embodiment 2, (2) tiie corre- 
spondence between the various levels of kvightness 

20 and times is established, (3) ttie pixels having tiie bright- 
ness oonresponding to tiie cunrent time are scanned in 
tiie mob'on description image, and (4) all matching pix- 
els are highlighted. The result of these steps is that only 
the desired outiine stands out. 

25 While there have been described what are at 
present conskiered to be preferred embodiments of tiie 
invention. It will be understood thert various modiftea- 
tions may be made thereto, and It is intended tiiat the 
appended claims cover all such modifications as fall 

30 witiiin tiie tme spirit and scope of the invention. 

Clainis 

1 . A representative Image display metiiod comprising: 

35 . 

a scene partition process for partitioning into 
scenes a motion image comprising a sequence 
of multiple images. 

a representative Image selection process for 
40 selecting a representative image to'repreeent 

each partitioned scene. 

a change detection process lor detecting 
changes occuning in an object witiiin each par- 
titioned scene, 

45 a change Information hokJ process for associ- 

ating and hokiing information related to the 
detected changes for each scene witti the rep- 
resentative image of the scene, and 
a display process for combining and displaying, 

so for at least one scene, tiie representative 

image of the scene with its associated change 
information. 

2. A representative image display apparatus compris- 
55 ing: 

an image input section (1 1 1) for inputting and 
storing^a motion image corrprising a sequence 
of multiple images. 
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a scene partition section (121) for analyzing 
the input motion image, detecting transition 
points in the scenes, and partitioning the 
motion image into scenes, 
a representative image selection section (122) 
for selecting a proper image from the motion 
image comprising multiple scenes and setting it 
as a representative image, 
a change inforntation hold section (120, 130) 
for deteclirig changes occum'ng in an object in 
each scene and associating and holding the 
information related to the change for each 
scene with the representative image of the 
scene, and 

a display control section (114) for coml»ning 
and displaying, for at least one scene, the rep- 
resentative image of the scene with its assod- 
ated change information. 

3. The representative image display apparatus 
according to claim 2. wherein changes occurring in 
the object are caused by movement of the object, 
and the change information hold section (120, 130) 
detects the movement of the object based on a dif- 
ference among a plurality of images, and the dis- 
play control section (114) displays text which 
indicates the movement of the object included in 
the images when it displays the representative 
image. 

4. The representative image display apparatus 
according to claim 2, wherein changes occurring in 
the object are caused by movement of the object, 
and the change information hold section (120. 130) 
detects the movement of the object based on a dif- 
ference among a plurality of inrtages. and further 
includes a motion description image creation sec- 
tion (123) which CTeates an image visually showing 
the movement of the object and the display control 
section (1 1 4) combines and displays the represent- 
ative image and the motion description image. 

5. The representative image display apparatus 
according to daim 4. wherein said motion descrip- 
tion image creation section (123) fetches, at a pre- 
set time interval, the image stored in the image 
input section and creates its outGne image, and 
superimposes the outfine image onto a prevfously 
created motion descriptfon image to repeatedly cre- 
ate a new motion description image. 



according to daims 4 through 6, wherein said dis- 
play control section (114) highlights the position of 
an object in a niotion description image at a sped- 
fied time, when an aribitrary time is specified 
5 between the start time and end time of a scene. 

8. The representative image display apparatus 

according to daim 7, wheirein said display control 
section (1 14) scans for pixels having a color con^e- 
w spending to the given time, and highlights the loca- 
tion of tiie ot^ject in a motion description image by 
changing the color of all the pixels found by scan- 
ning. 

IS 9. The representative image display apparatus 
according to claim 2, wherein changes occurring in 
the object are changes in color of the object, and 
the change information hold section (120, 130) 
detects a change in color of the object based on dif- 

20 ferences among a plurality of images and the dis- 
play contrd section cGsplays text which indicates 
the change in color of the object when said display 
control sedion (114) the representative image is 
displayed. 

25 

ia A motion image search apparatus for performing 
searches for a motion image, comprising: 

the representative image display apparatus 
30 according to claims 2 ttirough 9. 

a video source (101) for playing back and sup- 
plying a motion image to said representative 
image display apparatus, 
a video source control section (1 1 2) for control- 
35 ling the playt>ack of the motion image from the 

video source, and 

a command input section (103) for inputting 
commands for search processing, 

wherein the video source control section 
40 (112) performs, when one scene from dis- 

played scenes and a playt)ack start time for ttie 
scene are specified 1^ a command, playback of 
the nxytion image from the playt)ack start time 
of the scene based on tiie convnand. 

45 



SO 



6. The representative image display apparatus 
according to daim 5, wherein said motion descrip- 
tion image creation section (123) creates new 
motion description images while changing colors of ss 
the outiines k>ased on the elapsed time from the 
start time of the scene. 

7. The representative image display apparatus 



8 



EP0 788 064 A2 



CO 



CM 



UIO 

op 



O 

LU 
CO 

>- 

Q. 
CO 







UJ 








LU 

i 


ION 




g 




CO 


a. 









CM 



CO 



o 
I— 

coo 



8 



III O 

ZD LU 

oo 

LU DC 

=^8 



>OURCE 






IMAGE 
INPUT SECTION 


video; 







o 

LU 
CO 

O 



o 



3CO 



UJ ^ p o 

CO 2 o 

LU ffi O 

Q- > LU CO 

ixiF=co 
cc 



sis 

CoSr 
^<CO 



CO 
CM 



CM 
CM 



CM 



O 

§ 

CO 
UJ 

o 



o 

I— 

CO 

8 

i 



:0 



UJ S3 



;c3 



UJ 7 

lis 



C3 

ujQ^g 

sill 



iZ 



.CO 
CO 



.CM 
CO 



CO 



o 

CM 



o 

CO 



9 



EP0788064A2 



A 



B 



C 




A1 A2 B1 B2 CI C2 C3 



Fig. 2 





VIDEO 
CAPTURE BOARD 



VIDEO PUYBACK APPARATUS 

zz 

6 



PERSONAL COMPUTER 



COLOR PRINTER 



Fig. 3 



10 



EP0788064A2 



201 

203-- 



OUTLINE IMAGE 
CREATION SECTION 




i 


COMPOSITE IMAGE 
CREATION SECTION 




I 


OUTLINE IMAGE 
STORAGE SECTION 





Fig. 4 



(START) 



GENERATE OUTUNE IMAGE 
FROM INPUT IMAGE 



I 



COMPOSE OUTUNE IMAGE 



I 



STORE COMPOSED IMAGE 
TO OUTLINE IMAGE 
STORAGE SECTION 



S211 



S212 



S213 



( END ) 



Fig. 5 



11 



EP0788064 A2 



156 157 

{ L- S 






START TIME :00K)0:00:00 
END TIME : 00:02:05:20 


A 


O 




START TIME :00K)2:05:21 
END TIME :00:02:29K)8 




O 




START TIME: 00:02:29:09 
END TIME : 00:03:16:15 


V 


ISCENEPARnnONSrARTl H ■! 


152^ 


15^3 

Fig. 6 



12 



EP 0788 064 A2 




13 



EP0788064A2 




8 



123 



201 
301 
202 
203 



OUTLINE IMAGE 
CREATION SECTION 



I 



HUE SETTING 
SECTION 



COMPOSITE IMAGE 
CREATION SECTION 



I 



OUTLINE IMAGE 
STORAGE SECTION 



Fig. 9 



14 



EP0788064 A2 



CSTART) 



GENERATE OUTUNE IMAGE 
FROM INPUT IMAGE 



VARY HUE FOR OUTLINE 
IMAGE 



S211 



S312 



COMPOSE OUTLINE IMAGE 



S212 



STORE COMPOSED IMAGE 
TO OUTLINE IMAGE 
STORAGE SECTION 



(JND) 



S213 



Fig. 10 



Fig. 11 



15 



EP0788064A2 



121 
123 



120 

s 

CATALOG CREATION SECTION 



SCENE 
PARTITION 
SECTION 




REPRESENTATIVE 
IMAGE CREATION 
SECTION 


MOTION 
DESCRIPTION IMAGE 
CREATION SECTION 


TIME 
CALCUUT10N 
SECTION 



Fig. 12 



122 
401 



130 



131 
133 



CATALOG STORAGE SECTION 


SCENE PARTITION 
. DATASTORAGE 
SECTION 


REPRESENTATIVE 
IMAGE STORAGE - 
SECTION 


MOTION 
'DESCRIPnON IMAGE 
STORAGE SECTION 


TIME 

STORAGE SECTION ' 









132 
-.411 



Fig. 13 



16 



EP07S80MA2 



155- 



151 



156 



o 



o 



157 



HIGHUGHTED DISPLAY 




START TIME 
END TIME 
CURRENT TIME 



00:00:00:00 
00:02:05:20 
00:01:02:28 



START TIME 
END TIME 
CURRENT TIME 



00:02:05:21 
00:02:29:08 
00:0220:00 



START TIME : 00:02:29:09 
END TIME : 00:03:16:15 
CURRENT TIME :00:03K)1:20 



iSCENEPARTinONSTARflWIil 



SUBSEQUEMTTMEj 



152 



153 451 452 



158 



Fig. 14 



17 



